Understanding Model-Based Reinforcement Learning and its Application in Safe Reinforcement Learning

Model-based reinforcement learning algorithms have been shown to achieve successful results on various continuous control benchmarks, but the understanding of model-based methods is limited. We try to interpret how model-based method works through novel experiments on state-of-the-art algorithms with an emphasis on the model learning part. We evaluate the role of the model learning in policy optimization and propose methods to learn a more accurate model. With a better understanding of model-based reinforcement learning, we then apply model-based methods to solve safe reinforcement learning (RL) problems with near-zero violation of hard constraints throughout training. Drawing an analogy with how humans and animals learn to perform safe actions, we break down the safe RL problem into three stages. First, we train agents in a constraint-free environment to learn a performant policy for reaching high rewards, and simultaneously learn a model of the dynamics. Second, we use model-based methods to plan safe actions and train a safeguarding policy from these actions through imitation. Finally, we propose a factored framework to train an overall policy that mixes the performant policy and the safeguarding policy. This three-step curriculum ensures near-zero violation of safety constraints at all times. As an advantage of model-based method, the sample complexity required at the second and third steps of the process is significantly lower than model-free methods and can enable online safe learning. We demonstrate the effectiveness of our methods in various continuous control problems and analyze the advantages over state-of-the-art approaches

Low temperature and temperature decline increase acute aortic dissection risk and burden: A nationwide case crossover analysis at hourly level among 40,270 patients.

Background: Acute aortic dissection (AAD) is a life-threatening cardiovascular emergency with high mortality, so identifying modifiable risk factors of AAD is of great public health significance. The associations of non-optimal temperature and temperature variability with AAD onset and the disease burden have not been fully understood. Methods: We conducted a time-stratified case-crossover study using a nationwide registry dataset from 1,868 hospitals in 313 Chinese cities. Conditional logistic regression and distributed lag models were used to investigate associations of temperature and temperature changes between neighboring days (TCN) with the hourly AAD onset and calculate the attributable fractions. We also evaluated the heterogeneity of the associations. Findings: A total of 40,270 eligible AAD cases were included. The exposure-response curves for temperature and TCN with AAD onset risk were both inverse and approximately linear. The risks were present on the concurrent hour (for temperature) or day (for TCN) and lasted for almost 1 day. The cumulative relative risks of AAD were 1.027 and 1.026 per 1°C lower temperature and temperature decline between neighboring days, respectively. The associations were significant during the non-heating period, but were not present during the heating period in cities with central heating. 23.13% of AAD cases nationwide were attributable to low temperature and 1.58% were attributable to temperature decline from the previous day. Interpretation: This is the largest nationwide study demonstrating robust associations of low temperature and temperature decline with AAD onset. We, for the first time, calculated the corresponding disease burden and further showed that central heating may be a modifier for temperature-related AAD risk and burden. Funding: This work was supported by the National Natural Science Foundation of China (92043301 and 92143301), Shanghai International Science and Technology Partnership Project (No. 21230780200), the Medical Research Council-UK (MR/R013349/1), and the Natural Environment Research Council UK (NE/R009384/1)

Lipid-Head-Polymer-Tail Chimeric Nanovesicles

Lipid nanovesicles (LNVs) and polymer nanovesicles (PNVs), also known as liposomes and polymersomes, are becoming increasingly vital in global health. One recent example is the widely distributed mRNA Covid-19 vaccines. However, the two major classes of nanovesicles both exhibit their own issues that significantly limit potential applications. Here, by covalently attaching a naturally occurring phosphate “lipid head” and a synthetic polylactide “polymer tail” via facile ring-opening polymerization on a 500-gram scale, a type of “chimeric” nanovesicles (CNVs) can be easily produced. Compared to LNVs, the reported CNVs exhibit reduced permeability for small and large molecules; on the other hand, the CNVs are less hydrophobic and exhibit enhanced tolerance toward proteins in buffer solutions without the need for hydrophilic polymeric corona such as poly(ethylene glycol), in contrast to conventional PNVs. The proof-of-concept in vitro delivery experiments using hydrophilic solutions of fluorescein-PEG, rhodamine-PEG, and anti-cancer drug doxorubicin demonstrate that these CNVs, as a structurally diverse class of nano-materials, are highly promising as alternative carriers for therapeutic molecules in translational nanomedicine

Tumor-intrinsic IRE1α signaling controls protective immunity in lung cancer

The IRE1α-XBP1 arm of the unfolded protein response (UPR) has been associated with immunosuppression and cancer progression. Here the authors show that IRE1α-XBP1 activation is associated with poor overall survival in patients with non-small cell lung cancer and that IRE1α loss in cancer cells promotes anti-tumor immune responses in lung cancer preclinical models